Graphical user interface marking feedback

ABSTRACT

A method for providing user feedback for a website includes generating, via a programming interface of a first computing device, a substantially transparent overlay for a graphical user interface (GUI) of a web page, the substantially transparent overlay preventing user interaction with the GUI of the web page, receiving coordinates of an area on the substantially transparent overlay, translating, by the first computing device, the coordinates of the area on the substantially transparent overlay to source code for an element on the GUI of the web page located at a corresponding position of the coordinates of the area on the substantially transparent overlay, and transmitting, by the first computing device, source code for the element to a second computing device for rendering the element on a second computing device GUI.

CROSS-REFERENCE TO RELATED APPLICATION

This is a continuation of co-pending U.S. Pat. Application No.17/676,959, filed Feb. 22, 2022, titled “Graphical User InterfaceMarking Feedback,” which is a continuation of U.S. Pat. Application No.16/923,327, filed Jul. 8, 2020, now U.S. Pat. No. 11,327,625, titled“Graphical User Interface Marking Feedback,” which claims priority toU.S. Provisional Pat. Application No. 62/875,189, filed on Jul. 17,2019, and titled “Graphical User Interface Marking Feedback,” thedisclosures of all of which are hereby incorporated by reference intheir entireties.

TECHNICAL FIELD

The present disclosure relates generally to providing user feedback fora website, and more particularly, but not exclusively, to providing agraphical user interface overlay for a webpage to receive user inputabout the web page.

BACKGROUND

Website users sometimes experience issues that result in the usersreporting problems with a web page or requesting assistance with the website (e.g., navigating the web site). Inputs for feedback reporting caninclude free-response text boxes and rating scales. These inputs may notaccurately capture interface specifics because the inputs rely on a userto describe the context in enough detail for a representative to be ableto visualize or recreate the situation. Users may not be familiar withsystem behavior or the source code powering the system, which can makeaccurate reporting difficult. Other techniques may require a user todownload or launch a screen-sharing program, or other extension.

SUMMARY

In one example, a method for providing user feedback for a website mayinclude generating, via a programming interface of a first computingdevice, a substantially transparent overlay for a graphical userinterface (GUI) of a web page. The substantially transparent overlay mayprevent user interaction with the GUI of the web page. The method mayfurther include receiving coordinates of an area on the substantiallytransparent overlay, translating, by the first computing device, thecoordinates of the area on the substantially transparent overlay tosource code for an element on the GUI of the web page located at acorresponding position of the coordinates of the area on thesubstantially transparent overlay, and transmitting, by the firstcomputing device, source code for the element to a second computingdevice for rendering the element on a second computing device GUI.

In another example, a system may include a first computing device, and asecond computing device configured to communicate with the firstcomputing device. The first computing device may be configured toperform operations including generating, via a programming interface ofthe first computing device, a substantially transparent overlay for agraphical user interface (GUI) of a web page. The substantiallytransparent overlay may prevent user interaction with the GUI of the webpage. The first computing device may be further configured to performoperations including receiving coordinates of an area on thesubstantially transparent overlay, translating the coordinates of thearea on the substantially transparent overlay to source code for anelement on the GUI of the web page located at a corresponding positionof the coordinates of the area on the substantially transparent overlay,and transmitting, by the first computing device, the source code for theelement to the second computing device for rendering the element on asecond computing device GUI.

In a further example, a non-transitory computer readable medium mayinclude instructions that are executable for performing operationsincluding generating, via a programming interface of a first computingdevice, a substantially transparent overlay for a graphical userinterface (GUI) of a web page. The substantially transparent overlay mayprevent user interaction with the GUI of the web page. The instructionsmay further include receiving coordinates of an area on thesubstantially transparent overlay, translating, by the first computingdevice, the coordinates of the area on the substantially transparentoverlay to source code for an element on the GUI of the web page locatedat a corresponding position of the coordinates of the area on thesubstantially transparent overlay; and transmitting, by the firstcomputing device, the source code for the element to a second computingdevice for rendering the element on a second computing device GUI.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a communication path for implementing agraphical user interface (GUI) marking tool according to some aspects ofthe present disclosure.

FIG. 2 is a block diagram of an example of a computing environment thatincludes a computing device suitable for use with a GUI marking toolaccording to some aspects of the present disclosure.

FIG. 3 is a flowchart illustrating an example of a method for providingGUI marking feedback with asynchronous communication according to someaspects of the present disclosure.

FIG. 4 is a flowchart illustrating an example of a method for providingGUI marking feedback with synchronous communication according to someaspects of the present disclosure.

FIG. 5 is a diagram illustrating an example of synchronous communicationbetween a user computer terminal and an agent computer terminalaccording to some aspects of the present disclosure.

FIG. 6 is a diagram illustrating an example of synchronous communicationbetween an agent computer terminal and a user computer terminalaccording to some aspects of the present disclosure.

FIG. 7 is a diagram illustrating an example of sending a screenshot forstorage in a database using asynchronous communication according to someaspects of the present disclosure.

FIG. 8 is a diagram illustrating an example of synchronous communicationbetween a user computer terminal and an agent computer terminal forassisting in menu navigation according to some aspects of the presentdisclosure.

DETAILED DESCRIPTION

While certain embodiments are described, these embodiments are presentedby way of example only, and are not intended to limit the scope ofprotection. The apparatuses, methods, and systems described herein maybe embodied in a variety of other forms. Furthermore, various omissions,substitutions, and changes in the form of the example methods andsystems described herein may be made without departing from the scope ofprotection.

Websites are built from web pages. Web pages are documents, typicallycomposed in plain text interspersed with formatting instructions ofHypertext Markup Language (HTML) or extensible HTML (XHTML). Web pagesmay incorporate elements from other websites with suitable markupanchors that mark the beginning and the end of a hypertext link. Ahypertext link enables a user to navigate to a new web page by clickingon the link with a mouse. Web pages are accessed and transported withthe Hypertext Transfer Protocol (HTTP), which may optionally employencryption (e.g., HTTP Secure (HTTPS)) to provide security and privacyfor the user. A web browser may render web page content according to itsHTML markup instructions onto a computer display.

In some cases, when viewing a web page, incidents may occur about whicha user wishes to initiate a communication with the owner or operator(e.g., a customer service agent) of the website containing the web page.For example, the user may experience a broken link, an error message, amisspelled word, etc., or may be confused about how to navigate the website. In such cases, the user may initiate a communication action (e.g.,a chat, phone call, or feedback button press, etc.) with a receiver, forexample, a human agent, a robot, or a database. The communication may besynchronous, for example, a live chat or phone call during which theuser interacts with the receiver in real-time, or asynchronous in whichthe user sends the question or feedback for later review by thereceiver.

Initiating the communication action can also initiate the graphical userinterface (GUI) marking tool of the present disclosure. A GUI displaysinteractive visual objects that convey information, and representactions that can be taken by the user. When the GUI marking tool isinitiated, a substantially transparent graphical overlay of the currentweb page may be generated and a capture mode enabled. The substantiallytransparent graphical overlay may “grey-out” the web page while allowingthe web page including text, graphics, tables, etc., to be visible, butany hyperlinks or other selectable aspects on the web page may benonfunctional. The capture mode may enable a target screen elementselection by which the user (or the receiver during synchronouscommunication) may select an area of the current screen display, forexample by hovering or clicking and dragging with a mouse.

The target screen element information may be transmitted to the receiverand a visual interpretation of the target area may be rendered on thescreen display of the receiver. The screen display of the receiver maymirror the screen display of the user. In the case of asynchronouscommunication, the target screen element information may be transmittedto the receiver and stored, for example in a database on a server, forlater analysis by an analyst.

In some implementations, an overlay may be optional. JavaScript (JS) maybe used to intercept user interactions, such as mouse clicks, direct theuser interactions to the GUI marking tool, and stop propagation of theuser interactions to the default event handlers. In anotherimplementation, a Cascading Style Sheet (CSS) may be used to give anHTML element a style property that disables default pointer interactionwith that element. For example, with the style property enabled,clicking a link does not activate the link. This property may be appliedto every element on the web page during the selection phase. JavaScriptmay track user input behaviors, such as mouse pointer location, clicks,etc., based on a point of origin relative to a displayed document ofwindow, and highlight or add HTML/CSS properties to a selected elementof the web page.

Some examples of the GUI marking tool of the present disclosure canenable users to report the area of a GUI related to a problem withoutknowledge of the source code or operating states. Web page issuesobserved in the past, for example, an alert message that is no longer bevisible leaving a blank area on the screen, as well as present problemsand problems unrelated to user behavior (e.g., incorrectly spelledwords) may be reported using the GUI marking tool.

FIG. 1 is a diagram illustrating a communication path for implementingthe GUI marking tool according to some aspects of the presentdisclosure. Referring to FIG. 1 , a user device 110 may display awebpage. The user device 110 may be, for example, a personal computer, alaptop, a mobile phone or other mobile device, or another computingdevice capable of displaying the webpage. The user device 110 may becapable of communicating with other devices over the network 150. Theuser device 110 may communicate with a server 120 via the network 150.The network 150 may be, for example, the Internet or another network.The user device 110 may request the webpage from the server 120 and theserver 120 may serve the webpage to the user device 110 via the network150. The server 120 may mediate communication between the user device110 and an agent device 140 and may provide storage for data andtransactions between the user device 110 and the agent device 140.

The agent device 140 may be a computer terminal of an owner or operatorof the website or a computer terminal of a service representativeresponsible to maintain the website. The agent device 140 may be, forexample, a personal computer, a laptop, a mobile device, or anothercomputing device. The agent device 140 may be capable of communicatingwith other devices, including the server 120 and the user device 110,over the network 150. In some implementations, communication between theuser device 110, the agent device 140, and the server 120 may beperformed via a virtual private network (VPN) or peer-to-peer networkconnection.

According to some aspects of the present disclosure, a user mayexperience an issue with a web page displayed on the user device 110.The user may initiate a communication, for example a live chat, betweenthe user device 110 and the agent device 140 via the network 150 and theserver 120. Initiating the live chat may also initiate the GUI markingtool. The user may use the GUI marking tool to identify the problem areaof the GUI of the user device 110. Information related to the problemarea may be transmitted from the user device 110 to the agent device140. The transmitted information may be used to render a visualreproduction of the GUI of the user device 110 on a display of the agentdevice. The visual reproduction of the GUI may show the problem area asidentified by the user. The server 120 may also store the informationrelated to the problem area transmitted from the user device 110 forfurther analysis.

FIG. 2 is a block diagram of an example of a computing environment thatincludes a computing device 205 suitable for use with a GUI marking toolaccording to some aspects of the present disclosure. Any of the userdevice 110, the agent device 140, and the server 120 may be configuredsubstantially similar to the computing device 205.

The computing device 205 in the computing environment 200 may includeone or more processing units, cores, or processors 210, memory 215(e.g., RAM, ROM, or the like), internal storage 220 (e.g., magnetic,optical, solid state storage, or organic), and I/O interface 225, any ofwhich may be coupled on a communication mechanism or a bus 230 forcommunicating information or embedded in the computing device 205. Asused herein, reference to a processing unit, core, or processor mayinclude one processor or more than one processor.

The computing device 205 may be communicatively coupled to an input/userinterface 235 and an output device/interface 240. Either one or both ofthe input/user interface 235 and the output device/interface 240 may bea wired or wireless interface. The input/user interface 235 may includeany device, component, sensor, or interface, physical or virtual, thatcan be used to provide input (e.g., buttons, touch-screen interface,keyboard, a pointing/cursor control, microphone, camera, or the like) tothe computing device 205. The output device/interface 240 may include adisplay, television, monitor, printer, speaker, or the like. In someexample implementations, the input/user interface 235 and the outputdevice/interface 240 may be embedded with or physically coupled to thecomputing device 205. Examples of the computing device 205 may include,but are not limited to, mobile devices (e.g., tablets, notebooks,laptops, personal computers, portable televisions, radios, and thelike), and devices not designed for mobility (e.g., desktop computers,other computers, and the like).

The computing device 205 may be communicatively coupled (e.g., via theI/O interface 225) to an external storage device 245, a network 250 forcommunicating with any number of networked components, devices, andsystems, including one or more computing devices of the same ordifferent configuration, etc. The computing device 205 or any connectedcomputing device may be functioning as, providing services of, orreferred to as a server, client, thin server, general machine,special-purpose machine, or another label.

The I/O interface 225 may include, but is not limited to, wired andwireless interfaces using any communication or I/O protocols orstandards (e.g., Ethernet, 802.11x, Universal System Bus, WiMax, modem,a cellular network protocol, and the like) for communicating informationto and from at least all the connected components, devices, and networkin the computing environment 200. The network 250 may be any network orcombination of networks (e.g., the Internet, local area network, widearea network, a telephonic network, a cellular network, satellitenetwork, and the like).

The computing device 205 can use or communicate using computer-usable orcomputer-readable media, including transitory media and non-transitorymedia. Transitory media include transmission media (e.g., metal cables,fiber optics), signals, carrier waves, and the like. Non-transitorymedia include magnetic media (e.g., disks and tapes), optical media(e.g., CD ROM, digital video disks, Blu-ray disks), solid-state media(e.g., RAM, ROM, flash memory, solid-state storage), and othernon-volatile storage or memory.

The computing device 205 can be used to implement techniques, methods,applications, processes, or computer-executable instructions in someexample computing environments. Computer-executable instructions can beretrieved from transitory media, and stored on and retrieved fromnon-transitory media. The executable instructions may originate from oneor more of any programming, scripting, and machine languages (e.g., C,C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others).

The processor(s) 210 may execute under any operating system (OS) (notshown), in a native or virtual environment. One or more applications maybe deployed that a include logic unit 260, an application programminginterface (API) unit 265, an input unit 270, an output unit 275, and aninter-unit communication mechanism 295 for the different units tocommunicate with each other, with the OS, and with other applications(not shown). The described units and elements can be varied in design,function, configuration, or implementation and are not limited to thedescriptions provided.

In some example implementations, when information or an executioninstruction is received by the API unit 265, it may be communicated toone or more other units (e.g., the logic unit 260, the input unit 270,the output unit 275).

In some instances, the logic unit 260 may be configured to control theinformation flow among the units and direct the services provided by theAPI unit 265, the input unit 270, the output unit 275 in some exampleimplementations described above. For example, the flow of one or moreprocesses or implementations may be controlled by the logic unit 260alone or in conjunction with the API unit 265.

In accordance with some aspects of the present disclosure, software codemay be embedded in a client-side software-programming interface, such asa website or mobile application. The software code may enablecommunication between the server and agent device of the websiteoperator. The communication may be asynchronous or synchronous.

In the case of asynchronous communication, a user may initiate afeedback process by, for example, clicking on or otherwise selecting aselectable feature such as a feedback button overlayed on a portion of aGUI. Receiving the selection of the selectable feature may cause thecode for the GUI marking tool to initiate a selection process as part ofthe feedback process. The code for the GUI marking tool may cause theprocessor of the user device to generate a substantially transparentoverlay over the existing GUI elements of the webpage on the display ofthe user device. The substantially transparent overlay may be generatedusing a combination of JavaScript, HTML, and CSS.

The overlayed GUI may “grey-out” the web page while allowing the webpage including text, graphics, tables, etc., to be visible, butpreventing direct interaction by the user with the underlying GUI of thewebpage. Instructions on how to proceed, for example, instructions toselect an area causing an issue to be reported, may be displayed. Theuser may then select (i.e., mark) an area of the web page on which toprovide feedback.

The selection (or marking) procedure may be performed in different ways.For example, when the user hovers the mouse cursor for a specifiedperiod of time over an element in the underlying GUI, the processor ofthe user device may cause that element to be highlighted. When the userclicks the mouse on a highlighted element, that element is marked andthe highlight persists when the mouse cursor leaves that area. Inanother example, when the user performs a drag and release of the mouse,a shape that serves as the marker is rendered in the area of theinteraction based on the length and width of the drag. In still anotherexample, shapes of a predetermined size may be rendered by the processorof the user device at each location clicked by the user. The renderedshape may be, for example, a transparent box, circle, or other shapehaving a defined outline. Other ways of interacting with the code forthe GUI marking tool may be used, for example, but not limited to,styluses, motion-interactive interfaces, finger touch, keyboard,eye-trackers, ear-trackers, face trackers, and code commands. Theselection (marking) procedure may create new GUI elements on theoverlayed GUI.

In some implementations, the overlay and capture mode may be temporarilydisabled, allowing the user to manipulate the GUI into a correctrepresentative state for providing feedback, if the GUI was not alreadyin the desired state when the GUI marking tool was initiated. The GUImarking tool may allow the overlay and capture mode to be temporarilydisabled while a key on the keyboard is held down. For example, a userwants to capture an expanded menu, but the menu is not expanded when theGUI marking tool is initiated, the user may hold down a keyboard key,expand the menu, and release the keyboard key. When the key is released,the overlay and capture mode may enabled and return to their last activestate with the menu expanded.

To create the overlayed GUI elements, the code for the GUI marking toolmay monitor the user device for interaction behaviors from a user, suchas mouse hovers or mouse clicks. The interaction behaviors may indicatethat a marking procedure is being performed. Based on a determinationthat a selection (marking) procedure is being performed, the code forthe GUI marking tool may create and render new GUI elements on theoverlay at the locations of the interactions to indicate whereinteraction behaviors occurred. One or more new GUI elements thatoverlay the existing GUI may be created and rendered on the display ofthe user device. The user device (e.g., the processor) may receive andstore in memory the input from the selection procedure indicating thelocations, for example the X and Y coordinates of the rendered GUIelements on the overlay.

The code for the GUI marking tool may cause the user device to monitorfor a concluding event that indicates the marking procedure is complete.A concluding event can be generated in multiple ways. For example, aftermarking a specified number of locations, the concluding event couldautomatically be generated and sent by the code for the GUI markingtool. In some implementations, the code for the GUI marking tool maygenerate a user-interactive feature, such as a button, that whenselected sends a concluding event that is received by the processor ofthe user device indicating that the selection process is complete.

Upon conclusion of the marking procedure (e.g., after the user devicereceives the concluding event indication), the code for the GUI markingtool may cause the user device to initiate a translation process totranslate the stored input indicating the locations of the rendered GUIelements on the overlay to corresponding elements on the GUI of theoriginal web page. For example, a software library function or anothermeans may be used to determine the coordinates of the elements on theoriginal web page GUI. The processor or other logic unit of the userdevice can then correlate the coordinates of the original web pageelements with the new GUI elements rendered on the overlay to determinethe web page elements selected during the selection process.

The code for the GUI marking tool may cause the processor or other logicunit of the user device to identify GUI source code, such as HTML tagsin a webpage, corresponding to the same X and Y coordinates and in somecase at different depths on the z axis, for example behind a modaloverlay of the web page GUI, as the marked locations, and may recordinformation about the identified GUI source code (i.e., a source coderecord), for example in a memory of the user device. Recording theinformation about the identified GUI source code can occur in multipleways. For example, an element of a webpage could be recorded as anentire HTML tag, an HTML id attribute of the HTML tag, or the entireHTML document with an additional CSS class added to the HTML tag.

Additionally or alternatively, the code for the GUI marking tool maycause the user device to generate one or more image files (e.g., .png,.jpeg, etc.) depicting the current state of the underlying GUI (withoutmarkers), the markers, or the markers overlaying the underlying GUI thatmay also be stored, at least temporarily, in the memory of the userdevice. For example, after rendering a new GUI elements, the code forthe GUI marking tool may cause the user device to generate a screenshotusing the HTML DOM and CSS styling (i.e., the Document Object Modelprogramming interface and Cascading Style Sheet styling) to render theimage.

The code for the GUI marking tool may cause the user device to transmitthe information to a remote server, for example via a network socket.The information may include, for example, but not limited to, the one ormore generated image files, the source code record, information aboutthe system including state-based information of the GUI, informationabout the user system (e.g., internet browser, application version,etc.), and information about the user (e.g., unique user identifiers,etc.). The visual representation can be complementary to a writtendescription provided by the user. The server may associate theinformation with a transaction (e.g., a user problem report) and maystore the information for later retrieval and analysis by an analyst.

In situations where the communication is synchronous (e.g., live chat,intelligent algorithm, etc.), the client-side (i.e., user) softwareprogramming interface may cause a processor of the user device tomonitor for an event to prompt the user device to initiate the feedbackprocess. The processor may be, for example, the central processing unit(CPU) or other logic unit of the user device. The event received by theprocessor may be, for example, but not limited to, an event initiated bya user feedback button, an event initiated by a remote agent, an eventimitated by a remote computer algorithm, etc.

For example, an agent or service representative (e.g., a live chatrepresentative) may need clarification on a portion of a web page wherea problem is occurring while discussing the problem with a user. Thelive chat representative may send an initiating event from their remotesystem to the client-side (i.e., user) live chat software, and the livechat software may send the initiating event to the GUI event handler.The GUI event handler may then notify the code for the GUI marking tooland the code for the GUI marking tool may cause processor or other logicunit of the user device to modify the web page as explained above withrespect to the asynchronous communication process.

In some implementations, a remote intelligent computer algorithm mayobserve user behavior and detect signs of a potential problem. Forexample, a user who experiences frustration with a broken link may clickthe link many times within a short time period. The remote intelligentcomputer algorithm may detect the multiple clicks, determine that thesequence of behavior is a situation where feedback is desired, andinitiate the feedback process. The remote intelligent computer algorithmmay send the initiating event directly to the code for the GUI markingtool through a network socket connection (i.e., a two-way communicationlink between two programs). Upon receiving a valid initiating event, theselection and translation processes may proceed as explained above withrespect to the asynchronous communication process.

After completion of the selection and translation processes, the codefor the GUI marking tool may cause the user device to transmit theinformation to a remote server, for example via a network socket. Theinformation may include, for example, but not limited to, the one ormore generated image files, the source code record, information aboutthe system including state-based information of the GUI, informationabout the user system (e.g., internet browser, application version,etc.), and information about the user (e.g., unique user identifiers,etc.). The server may associate the information with a transaction(e.g., a user problem report) and may store the information, for examplein a database. The server may also transmit the information to an agentdevice during synchronous communication between the user device and theagent device.

The code for the GUI marking tool on the receiving device (e.g., theagent device 140) may cause the receiving device to render arepresentation of the marked areas of the web page in real time. In somecases, the one or more generated image files of the web page may bereceived and the receiving device may display the received web pageimages. In some cases, the HTML source code for the marked up web page(i.e., the web page including the added GUI elements) may be receivedand rendered. In some cases, for example when the receiving device canaccess the web page from the server, only the HTML source code for thenew GUI element may transmitted by the user device and rendered by thereceiving device on the same web page.

In some implementations, similar marking, selection, and translationprocesses may be used to enable an agent to provide a modified web pageto a user for instructional purposes. For example, if a user is speakingto a support agent about a menu item that cannot be located, the supportagent could use the GUI marking process to mark the menu item and showthe user where it is located.

FIG. 3 is a flowchart illustrating an example of a method for providingGUI marking feedback with asynchronous communication according to someaspects of the present disclosure. Referring to FIG. 3 , at block 310, afeedback process involving a user device may be initiated. For example,a user may experience an issue with a web page and wish to providefeedback to the operator of the website. In some implementations, theuser may perform a selection action, for example a mouse click, on anindicated area, for example a “feedback” or “help” button displayed onthe GUI. For example, code for the GUI marking tool may cause the userdevice to generate and display the “feedback” or “help” button. The userdevice may receive the button press, for example via a mouse click, andcode for the GUI marking tool may cause the user device (e.g., via theprocessor or other logic unit) to initiate the feedback process.

At block 320, substantially transparent overlay of the GUI of the webpage may be generated. For example, when the feedback process isinitiated, the code for the GUI marking tool may cause the processor orother logic unit of the user device to generate the substantiallytransparent overlay on the GUI displayed on the user device. In someimplementations, the overlay may “grey-out” the web page while allowingthe web page including text, graphics, tables, etc., to be visible, butpreventing direct interaction by the user with the underlying GUI of thewebpage. Instructions may be displayed to the user via the overlayed GUIto select an area of the web page related to the issue.

At block 330, the user may select an area of the web page. For example,the user may hover the mouse cursor over an element in the underlyingGUI to highlight the area and click the mouse on the highlighted elementto select it, perform a drag and release of the mouse to cause a shapeto be rendered in the area, perform a mouse click to cause shapes of apredetermined size to be rendered at the location clicked by the user,etc. Other ways of selecting an area may be employed, for example, butnot limited to, using styluses, motion-interactive interfaces, fingertouch, keyboard, eye-trackers, ear-trackers, face trackers, and codecommands.

To create the overlayed GUI elements, the code for the GUI marking toolmay monitor the user device for interaction behaviors from a user, suchas mouse hovers or mouse clicks. The interaction behaviors may indicatethat a marking procedure is being performed. Based on a determinationthat a selection (marking) procedure is being performed, the code forthe GUI marking tool may create and render new GUI elements on theoverlay at the locations of the interactions to indicate whereinteraction behaviors occurred. One or more new GUI elements thatoverlay the existing GUI may be created and rendered on the display ofthe user device. The user device (e.g., the processor) may receive andstore in memory the input from the selection procedure indicating thelocations, for example the X and Y coordinates of the rendered GUIelements on the overlay.

At block 340, an indication that the selection process is complete maybe generated. The code for the GUI marking tool may cause the userdevice to monitor for a concluding event that indicates the markingprocedure is complete. A concluding event can be generated in multipleways. For example, after marking a specified number of locations, theconcluding event could automatically be generated and sent by the codefor the GUI marking tool. In some implementations, the code for the GUImarking tool may generate a user-interactive feature, such as a button,that when selected sends a concluding event that is received by theprocessor of the user device indicating that the selection process iscomplete.

At block 350, the coordinates of the selection information may betranslated to the coordinates of the underlying web page. The code forthe GUI marking tool may cause the user device to initiate a translationprocess to translate the stored input indicating the locations of therendered GUI elements on the overlay to corresponding elements on theGUI of the original web page. For example, a software library functionor another means may be used to determine the coordinates of theelements on the original web page GUI. The processor or other logic unitof the user device can then correlate the coordinates of the originalweb page elements with the new GUI elements rendered on the overlay todetermine the web page elements selected during the selection process.

The code for the GUI marking tool may cause the processor or other logicunit of the user device to identify GUI source code, such as HTML tagsin a webpage, corresponding to the same X and Y coordinates and in somecase at different depths on the z axis, for example behind a modaloverlay of the web page GUI, as the marked locations, and may recordinformation about the identified GUI source code (i.e., a source coderecord), for example in a memory of the user device. Recording theinformation about the identified GUI source code can occur in multipleways. For example, an element of a webpage could be recorded as anentire HTML tag, an HTML id attribute of the HTML tag, or the entireHTML document with an additional CSS class added to the HTML tag.

Additionally or alternatively, the code for the GUI marking tool maycause the user device to generate one or more image files (e.g., .png,.jpeg, etc.) depicting the current state of the underlying GUI (withoutmarkers), the markers, or the markers overlaying the underlying GUI thatmay also be stored, at least temporarily, in the memory of the userdevice. For example, after rendering a new GUI elements, the code forthe GUI marking tool may cause the user device to generate a screenshotusing the HTML DOM and CSS styling (i.e., the Document Object Modelprogramming interface and Cascading Style Sheet styling) to render theimage.

At block 360, the selection information may be transmitted to theserver. The code for the GUI marking tool may cause the user device totransmit the information to a remote server, for example via a networksocket. The information may include, for example, but not limited to,the one or more generated image files, the source code record,information about the system including state-based information of theGUI, information about the user system (e.g., internet browser,application version, etc.), and information about the user (e.g., uniqueuser identifiers, etc.). The visual representation can be complementaryto a written description provided by the user.

At block 370, the selection information may be stored on the server. Theserver may associate the information with a transaction (e.g., a userproblem report) and may store the information in a database in internalor external storage for later retrieval and analysis by an analyst.

FIG. 4 is a flowchart illustrating an example of a method 400 forproviding GUI feedback with synchronous communication according to someaspects of the present disclosure. Referring to FIG. 4 , at block 410 auser may experience an issue with a web page and wish to obtainassistance from the operator of the website. In some implementations,the user may be in communication with a live chat agent. In someimplementations, the user may be in communication with an intelligentalgorithm executing on a server. During the communication session, theclient-side (i.e., user) software-programming interface may cause aprocessor of the user device to monitor for an event to prompt the userdevice to initiate the feedback process. The live chat agent orintelligent algorithm may generate the event and transmit the initiatingevent to the user device.

At block 415, the transmitted initiating event may be received by theprocessor of the client device. The initiating event may be transmittedfrom the remote system may be received by the client-side (i.e., user)live chat software, and the live chat software may send the initiatingevent to the GUI event handler. The GUI event handler may then notifythe code for the GUI marking tool and the code for the GUI marking toolmay cause the processor or other logic unit of the user device toinitiate the feedback process at the user device.

At block 420, a substantially transparent overlay of the GUI of the webpage may be generated. For example, when the feedback process isinitiated, the code for the GUI marking tool may cause the processor orother logic unit of the user device to generate the substantiallytransparent overlay on the GUI displayed on the user device. In someimplementations, the overlay may “grey-out” the web page while allowingthe web page including text, graphics, tables, etc., to be visible, butpreventing direct interaction by the user with the underlying GUI of thewebpage. Instructions may be displayed to the user via the overlayed GUIto select an area of the web page related to the issue.

At block 425, the user may select an area of the web page. For example,the user may hover the mouse cursor over an element in the underlyingGUI to highlight the area and click the mouse on the highlighted elementto select it to cause a shape to be rendered, perform a drag and releaseof the mouse to cause a shape to be rendered in the area, perform amouse click to cause shapes of a predetermined size to be rendered atthe location clicked by the user, etc. The rendered shape may be, forexample, a transparent box, circle, or other shape having a definedoutline. Other ways of selecting an area may be employed, for example,but not limited to, using styluses, motion-interactive interfaces,finger touch, keyboard, eye-trackers, ear-trackers, face trackers, andcode commands.

To create the overlayed GUI elements, the code for the GUI marking toolmay monitor the user device for interaction behaviors from a user, suchas mouse hovers or mouse clicks. The interaction behaviors may indicatethat a marking procedure is being performed. Based on a determinationthat a selection (marking) procedure is being performed, the code forthe GUI marking tool may create and render new GUI elements on theoverlay at the locations of the interactions to indicate whereinteraction behaviors occurred. One or more new GUI elements thatoverlay the existing GUI may be created and rendered on the display ofthe user device. The user device (e.g., the processor) may receive andstore in memory the input from the selection procedure indicating thelocations, for example the X and Y coordinates of the rendered GUIelements on the overlay.

At block 430, an indication that the selection process is complete maybe generated. The code for the GUI marking tool may cause the userdevice to monitor for a concluding event that indicates the markingprocedure is complete. A concluding event can be generated in multipleways. For example, after marking a specified number of locations, theconcluding event could automatically be generated and sent by the codefor the GUI marking tool. In some implementations, the code for the GUImarking tool may generate a user-interactive feature, such as a button,that when selected sends a concluding event that is received by theprocessor of the user device indicating that the selection process iscomplete.

At block 435, the coordinates of the selection information may betranslated to the coordinates of the underlying web page. The code forthe GUI marking tool may cause the user device to initiate a translationprocess to translate the stored input indicating the locations of therendered GUI elements on the overlay to corresponding elements on theGUI of the original web page. For example, a software library functionor another means may be used to determine the coordinates of theelements on the original web page GUI. The processor or other logic unitof the user device can then correlate the coordinates of the originalweb page elements with the new GUI elements rendered on the overlay todetermine the web page elements selected during the selection process.

The code for the GUI marking tool may cause the processor or other logicunit of the user device to identify GUI source code, such as HTML tagsin a webpage, corresponding to the same X and Y coordinates and in somecase at different depths on the z axis, for example behind a modaloverlay of the web page GUI, as the marked locations, and may recordinformation about the identified GUI source code (i.e., a source coderecord), for example in a memory of the user device. Recording theinformation about the identified GUI source code can occur in multipleways. For example, an element of a webpage could be recorded as anentire HTML tag, an HTML id attribute of the HTML tag, or the entireHTML document with an additional CSS class added to the HTML tag.

Additionally or alternatively, the code for the GUI marking tool maycause the user device to generate one or more image files (e.g., .png,.jpeg, etc.) depicting the current state of the underlying GUI (withoutmarkers), the markers, or the markers overlaying the underlying GUI thatmay also be stored, at least temporarily, in the memory of the userdevice. For example, after rendering a new GUI elements, the code forthe GUI marking tool may cause the user device to generate a screenshotusing the HTML DOM and CSS styling (i.e., the Document Object Modelprogramming interface and Cascading Style Sheet styling) to render theimage.

At block 440, the selection information may be transmitted to theserver. The code for the GUI marking tool may cause the user device totransmit the information to a remote server, for example via a networksocket. The information may include, for example, but not limited to,the one or more generated image files, the source code record,information about the system including state-based information of theGUI, information about the user system (e.g., internet browser,application version, etc.), and information about the user (e.g., uniqueuser identifiers, etc.). The visual representation can be complementaryto a written description provided by the user.

At block 445, the selection information may be stored on the server. Theserver may associate the information with a transaction (e.g., a userproblem report) and may store the information in a database in internalor external storage. The server may also transmit the information to anagent device during synchronous communication between the user deviceand the agent device.

At block 450, the selection information may be transmitted to thereceiver device. The server may also transmit the information to anagent device during synchronous communication between the user deviceand the agent device.

At block 455, The code for the GUI marking tool on the receiving device(e.g., the agent device 140) may cause the receiving device to render arepresentation of the marked areas of the web page in real time. In somecases, the one or more generated image files of the web page may bereceived and the receiving device may display the received web pageimages. In some cases, the HTML source code for the marked up web page(i.e., the web page including the added GUI elements) may be receivedand rendered. In some cases, for example when the receiving device canaccess the web page from the server, only the HTML source code for thenew GUI element may transmitted by the user device and rendered by thereceiving device on the same web page.

It should be appreciated that the specific steps illustrated in FIGS. 3and 4 provide particular methods for providing graphical user interfacemarking feedback according to embodiments of the present invention.Other sequences of steps may also be performed according to alternativeembodiments. For example, alternative embodiments of the presentinvention may perform the steps outlined above in a different order.Moreover, the individual steps illustrated in FIGS. 3 and 4 may includemultiple sub-steps that may be performed in various sequences asappropriate to the individual step. Furthermore, additional steps may beadded or removed depending on the particular applications. For example,in some implementations, an overlay may be optional. JavaScript (JS) maybe used to intercept user interactions and stop propagation of the userinteractions to the default event handlers, or CSS may be used to givean HTML element a style property that disables default pointerinteraction with that element. One of ordinary skill in the art wouldrecognize many variations, modifications, and alternatives.

The methods 300 and 400 may be embodied on a non-transitory computerreadable medium, for example, but not limited to, a memory of userdevice 110, agent device 140, or other non-transitory computer readablemedium, having stored therein a program including computer executableinstructions for making a processor, computer, or other programmabledevice execute the operations of the methods.

FIG. 5 is a diagram illustrating an example of synchronous communicationbetween a user device and an agent device according to some aspects ofthe present disclosure. As illustrated in FIG. 5 , using the feedbackprocess according to aspects of the present disclosure, selectioninformation, for example source code corresponding to the selected areaon the GUI of the user device 510, is transmitted by the user device 510and rendered on the agent device 520 to enable the agent to visualizethe GUI as seen by the user.

Using synchronous communication, the user of the user device 510 maycontrol the information seen by the agent on the agent device 520. Theagent device 520 may render a visualization of the source code elementstransmitted by the user device 510. For example, the user of the userdevice 510 may hover a cursor 512 over an area 514 of the display.Alternatively or additionally, the user may click on the area or clickand drag to encompass the area. The source code elements transmitted bythe user device 510 may cause the corresponding area 524 of the displayon the agent device 520 to appear highlighted, thus identifying an areaof the web page the user is seeking help with.

FIG. 6 is a diagram illustrating an example of synchronous communicationbetween an agent device and a user device according to some aspects ofthe present disclosure. As illustrated in FIG. 6 , similar marking,selection, and translation processes according to the feedback processof the present disclosure may be used to enable an agent to provide amodified web page to a user for instructional purposes.

Using synchronous communication, a user device 610 may render avisualization of the source code elements transmitted by an agent device620. For example, the agent may hover a cursor 622 over an area 624 ofthe display on the agent device 620. Alternatively or additionally, theagent may click on the area or click and drag to encompass the area. Thesource code elements transmitted by the agent device 620 may cause thecorresponding area 614 of the display on the user device 610 to appearhighlighted, thus enabling the agent to provide information orinstructions to the user regarding the highlighted portion of thedisplay.

FIG. 7 is a diagram illustrating an example of sending a screenshot forstorage in a server database using asynchronous communication accordingto some aspects of the present disclosure. As illustrated in FIG. 7 ,the marking, selection, and translation processes according to thefeedback process of the present disclosure may be used for asynchronouscommunication with a website provider enabling the storage of theselection information in a database in a server for subsequent retrievaland analysis by an analyst (e.g., agent).

Referring to FIG. 7 , a user may select a target area 714 of a displayon the user device 710. For example, the user of the user device 710 mayhover a cursor 712 over a target area 714 of the display. Alternativelyor additionally, the user may click on the area or click and drag toencompass the area. Selecting the target area 714 may cause the targetarea 714 to be highlighted and may cause a screenshot image of thedisplay to be generated. Alternatively, an additional action, forexample, a right-click of the mouse or a button press, may cause thescreenshot to be generated. The generated screenshot image may be storedin a database for future analysis. At a later time, an agent (e.g.,analyst) may use an agent terminal 720 to access the screenshot imagefrom the database. The screenshot image, including the highlightedtarget area 714 to be reproduced as a highlighted area 724 on the agentterminal 720. The screenshot image may enable the analyst to identifythe area (e.g. the highlighted area of the display) that the userrequested information about and provide the appropriate feedback to theuser.

FIG. 8 is a diagram illustrating an example of synchronous communicationbetween a user device and an agent device for assisting in menunavigation according to some aspects of the present disclosure. Asillustrated in FIG. 8 , each step required to navigate a menu may bemarked, selected, translated, and transmitted for rendering on an agentdevice to enable the agent to walk a user through the steps required forproper navigation of the menu.

Referring to FIG. 8 , a user may hover a cursor 812 over a target areaof the display of the user device 810. The target area may be a menu 814having one or more sub-menus 816 (e.g., a complex menu). Hovering thecursor 812 over a particular selection of a menu 814 may cause a relatedsub-menu 816 to open. In some implementations, when a user wants tocapture an expanded menu, but the menu is not expanded when the GUImarking tool is initiated, the user may hold down a keyboard key, expandthe menu, and release the keyboard key. Source code elementscorresponding to the selected menus 814 and sub-menus 816 on the GUI ofthe user device 810 may be transmitted by the user device 810 andrendered as menus 824 and sub-menus 826 on the GUI of on the agentdevice 820, thereby enabling the agent to visualize the GUI as the usermoves through the menus and sub-menus. The agent may then provideguidance to the user regarding menu navigation.

The features discussed herein are not limited to any particular hardwarearchitecture or configuration. A computing device can include anysuitable arrangement of components that provide a result conditioned onone or more inputs. Suitable computing devices include multipurposemicroprocessor-based computer systems accessing stored software thatprograms or configures the computing system from a general-purposecomputing apparatus to a specialized computing apparatus implementingone or more aspects of the present subject matter. Any suitableprogramming, scripting, or other type of language or combinations oflanguages may be used to implement the teachings contained herein insoftware to be used in programming or configuring a computing device.

The examples and embodiments described herein are for illustrativepurposes only. Various modifications or changes in light thereof will beapparent to persons skilled in the art. These are to be included withinthe spirit and purview of this application, and the scope of theappended claims, which follow.

What is claimed is:
 1. A method for providing user feedback for awebsite using a graphical user interface (GUI), the method comprising:receiving, by a client device, an initiating event; in response to theinitiating event, generating, via a programming interface of the clientdevice, a substantially transparent overlay for a GUI of a web page, thesubstantially transparent overlay preventing user interaction with theGUI of the web page, the substantially transparent overlay configured toaccept a selection of an area of the web page; receiving, by the clientdevice, the selection of an area of the web page on the substantiallytransparent overlay, wherein selection information comprises at leastone of: generated image files, a source code record, state-basedinformation of the GUI, information about a user system, or informationabout a user; rendering, by the client device, web page elements fromthe selected area on the substantially transparent overlay; in responseto detecting a concluding event for the selection, translating, by theclient device, coordinates of the rendered web page elements on thesubstantially transparent overlay to corresponding elements on the GUIof the web page; identifying, by the client device, GUI source codecorresponding to coordinates of the selected web page elements on theGUI of the web page; creating an image file including the GUI of the webpage and a graphical element rendered on the substantially transparentoverlay, the image file created from source code of the GUI of the webpage and the graphical element; transmitting, by the client device, thesource code and selection information for the selected web page elementto a remote server; storing the source code and selection information onthe remote server; and associating the source code and selectioninformation stored on the remote server with a user problem report andstoring the user problem report, the source code for the element, andthe selection information in a database on the remote server.
 2. Themethod of claim 1, wherein the remote computing device is configured torender the elements of the GUI of the web page defined by the sourcecode on a GUI of the remote computing device.
 3. The method of claim 1,wherein the initiating event is generated by the remote computing deviceand transmitted to the client device.
 4. The method of claim 1, whereincoordinates of the area on the substantially transparent overlay arereceived via a user input device.
 5. The method of claim 1, wherein thecoordinates of the selected web page elements comprise X and Ycoordinates at different depths on a Z axis of the web page GUI.
 6. Themethod of claim 1, further comprising accessing, by a remote computingdevice, the source code and selection information from the remote serverat a subsequent time; and rendering, by the remote computing device, arepresentation of the selected area of the web page.
 7. A system,comprising: a first computing device; a second computing device; and aserver configured to communicate with the first computing device and thesecond computing device; the first computing device configured toperform operations including: receiving, from the second computingdevice, an initiating event; in response to the initiating event,generating, via a programming interface of the first computing device, asubstantially transparent overlay for a graphical user interface (GUI)of a web page, the substantially transparent overlay operable to preventuser interaction with the GUI of the web page, the substantiallytransparent overlay configured to accept a selection of an area of theweb page; receiving, by the first computing device, the selection of anarea of the web page on the substantially transparent overlay, whereinselection information comprises at least one of: generated image files,a source code record, state-based information of the GUI, informationabout a user system, or information about a user; rendering, by thefirst computing device, web page elements from the selected area on thesubstantially transparent overlay; in response to detecting a concludingevent for the selection, translating coordinates of the rendered webpage elements on the substantially transparent overlay to correspondingelements on the GUI of the web page; identifying, by the first computingdevice, GUI source code corresponding to the coordinates of the selectedweb page elements on the GUI of the web page; creating an image fileincluding the GUI of the web page and a graphical element rendered onthe substantially transparent overlay, the image file created fromsource code of the GUI of the web page and the graphical element;transmitting, by the first computing device, the source code andselection information for the selected web page element to the server;and storing the source code and selection information on the server,wherein the server is configured to associate the source code andselection information with a user problem report and store the userproblem report, the source code for the element, and the selectioninformation for the element in a database on the server.
 8. The systemof claim 7, wherein the second computing device is configured to renderthe elements of the GUI of the web page defined by the source code onthe web page GUI of the second computing device.
 9. The system of claim7, wherein the second computing device is configured to render theelements of the GUI of the web page defined by the source code on a GUIof the second computing device.
 10. The system of claim 7, wherein theinitiating event is generated by the second computing device andtransmitted to the first computing device.
 11. The system of claim 7,wherein the coordinates of the area on the substantially transparentoverlay are received via a user input device.
 12. The system of claim 7,wherein the coordinates of the selected web page elements comprise X andY coordinates at different depths on a Z axis of the web page GUI. 13.The system of claim 7, wherein the second computing device configured toaccess the source code and selection information from the server at asubsequent time; and render a representation of the selected area of theweb page GUI.
 14. A non-transitory computer readable medium havingstored therein instructions that are executable for performingoperations including: receiving, by a client device, an initiatingevent; in response to the initiating event, generating, via aprogramming interface of the client device, a substantially transparentoverlay for a GUI of a web page, the substantially transparent overlaypreventing user interaction with the GUI of the web page, thesubstantially transparent overlay configured to accept a selection of anarea of the web page; receiving, from an input device, the selection ofan area of the web page on the substantially transparent overlay,wherein selection information comprises at least one of: generated imagefiles, a source code record, state-based information of the GUI,information about a user system, or information about a user; rendering,by the client device, web page elements from the selected area on thesubstantially transparent overlay; in response to detecting a concludingevent for the selection, translating, by the client device, coordinatesof the rendered web page elements on the substantially transparentoverlay to corresponding elements on the GUI of the web page;identifying, by the client device, GUI source code corresponding tocoordinates of the selected web page elements on the GUI of the webpage; creating an image file including the GUI of the web page and agraphical element rendered on the substantially transparent overlay, theimage file created from source code of the GUI of the web page and thegraphical element; and transmitting, by the client device, the sourcecode and selection information for the selected web page element to aremote server, wherein the server is configured to associate the sourcecode and selection information with a user problem report and store theuser problem report, the source code for the element, and the selectioninformation in a database on the server.
 15. The non-transitory computerreadable medium as defined in claim 14, wherein the coordinates of theselected web page elements comprise X and Y coordinates at differentdepths on a Z axis of the web page GUI.